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PREFACE 


Educational  evaluation  is  a  critical  function  for  all  training 
institutions.  It  is  necessary  to  determine  if  the  training  goals 
^re  being  met,  and  if  not,  why!  An  effective  evaluation  program 
must  determine  if  training  deficiencies  exist,  so  that  reasonable, 
valid  changes  can  be  implemented  to  improve  the  training  product. 
The  US  Army’s  training  system,  governed  by  TRADQC  Regulation  350-7, 
A  Systems  Approach  to  Training,  recognises  the  importance  of 
educational  evaluation.  As  defined  in  the  regulation,  the  purpose 
of  evaluation  is  "to  determine  if  soldiers  are  capable  of  perform¬ 
ing  their  duty  tasks  and  if  units  can  accomplish  their  mission." 
(4:3--l)  The  evaluation  mission  is  considered  a  shared  responsi¬ 
bility  between  the  training  base  and  field  units.  However,  the 
training  bases  are  uniquely  responsible  for  assessing  the  effec¬ 
tiveness  of  the  resident  and  nonresident  training  programs 
<4:3-2). 

The  US  Army  Aviation  Center  (USAAVNC) ,  Fort  Rucker,  Alabama, 
is  the  training  base  for  all  US  Army  aviation  related  training. 
Therefore,  it  is  responsible  for  the  evaluation  of  all  Army 
resident  and  nonresident  aviation  training  programs.  The  Direc¬ 
torate  of  Evaluation  and  Standardisation  <DES) ,  at  USAAVNC,  is 
charged  with  the  responsibility  of  evaluating  the  effectiveness  of 
training  at  the  Aviation  Center.  DES  performs  this  function 
through  its  Training  Evaluation  Strategy.  This  strategy,  which 
will  be  discussed  in  greater  length  in  Chapter  2,  establishes  a 
four-step  procedure  to  manage  the  evaluation  mission.  It  consoli¬ 
dates  feedback,  from  a  variety  of  separate  evaluation  programs,  to 
identify  training  related  problems  and  recommend  appropriate  cor¬ 
rective  actions.  One  source  providing  feedback  for  the  strategy 
is  the  Classroom  Training  Evaluation  <CTE)  Program. 

The  current  CTE  program  lacks  definitive  procedures  and  is  not 
providing  adequate  feedback  to  support  DES’s  overall  Training 
Evaluation  Strategy.  This  deficiency  was  catalyst  for  this  study 
to  develop  a  more  detailed  CTE  program,  thus  improving  the  quality 
of  classroom  evaluations  and  provide  better  feedback  to  the 
evaluation  strategy. 

It  must  be  noted  that  Chapter  4  of  this  report  is  not  printed 
in  the  standard  ACSC  study  format.  It  is  intended  to  be  a  "stand 
alone"  document  that  meets  the  format  required  by  the  study’s 
sponsor,  the  Directorate  of  Evaluation  and  Standard: zation,  US 
Army  Aviation  Center. 
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EXECUTIVE  SUMMARY 


^  Part  of  our  College  mission  is  distribution  of  the 
students’  problem  solving  products  to  DoD 
j^>  sponsors  and  other  interested  agencies  to 
enhance  insight  into  contemporary,  defense 
related  issues.  While  the  College  has  accepted  this 
‘  product  as  meeting  academic  requirements  for 
graduation,  the  views  and  opinions  expressed  or 
implied  are  solely  those  of  the  author  and  should 
not  be  construed  as  carrying  official  sanction. 
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85-1385 


AUTHOR(S) 
TITLE  A 


MAJOR  MICHAEL  J.  JUNEAU,  US  ARMY 


A  CLASSROOM  TRAINING  EVALUATION  PROGRAM  FOR  THE 
DIRECTORATE  OF  EVALUATION  AND  STANDARDIZATION, 
US  ARMY  AVIATION  CENTER  • 


I-  Purpose;  To  develop  a  Classroom  Training  Evaluation  Program 
■for  the  Directorate  of  Evaluation  and  Standardization  (DES) ,  US 
Army  Aviation  Center,  Fort  Rucker,  Alabama. 

II*  Problem:  The  DES  classroom  evaluation  program  was 

inadequate  and  did  not  meet  all  DES  responsibilities  and  require¬ 
ments.  Development  of  a  more  comprehensive  evaluation  program  was 
required  to  accomplish  these  needs. 

HI*  Discussion:  The  proposed  classroom  evaluation  program 
establishes  procedures  and  responsibilities  for  classroom  evalua¬ 
tions  at  the  Army  Aviation  Center.  It  was  developed  through  an 
examination  of  appropriate  policies  and  directives  to  determine 
DEL  requirements  for  classroom  evaluation  and  an  investigation  of 
local  staffing  guides,  as  well  as,  an  interview  with  DES  represen¬ 
tatives,  to  determine  resources  available  for  a  classroom  evalua¬ 
tion  program.  The  program  designates  the  Evaluation  Division  as 
the  responsible  agency  and  the  Internal  Evaluation  Branch,  through 
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a  program  coordinator,  as  the  office  responsible  for  the  day-to- 
day  management  of  the  program.  The  proposed  program  is  based  upon 
the  use  of  two  separate  types  of  classroom  evaluations.  Type  1 
classroom  evaluations  are  designed  to  provide  a  "snapshot  of 
training",  through  routine,  periodic,  classroom  observations,  by 
individuals  not  specially  trained  in  educational  evaluation.  Type 
1  evaluations  will  assist  DES  in  detecting  training  problems  in 
the  training  environment.  Type  2  classroom  evaluations  will  pro¬ 
vide  a  more  in-depth  analysis  of  training  problems  that  have  been 
previously  identified.  They  are  designed  to  determine  the  true 
nature  of  training  problems  and  identify  their  possible  causes. 
Type  2  evaluations  will  be  performed  by  personnel  specifically 
trained  in  education  evaluation  techniques. 

IV.  Conclusion;  The  current  classroom  evaluation  program  is  ' 
inadequate  to  meet  DES  classroom  evaluation  requirements.  The 
proposed  program  will  meet  most  of  DES’s  classroom  evaluation 
requirements  but  should  be  validated  to  "fine  tune"  the  final 
product  before  final  adoption. 

V.  Recommendation ;  The  Directorate  of  Evaluation  and  Stand¬ 
ardisation,  US  Army  Aviation  Center,  implement  the  proposed  class¬ 
room  evaluation  program,  and  validate  the  effectiveness  of  the 
program  through  its  use  at  Fort  Rucker. 
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Chapter  One 


INTRODUCTION 

t 

This  study  was  requested  by  the  Directorate  of  Evaluation  and 
Standardization  (DES),  US  Army  Aviation  Center.  This  chapter  will 
explain  the  parameters  established  by  DES,  which  shaped  the  scope 
and  detail  of  the  study. 


PURPOSE  OF  PROJECT 

This  project  develops  a  Classroom  Training  Evaluation  CCTE) 
Program  for  the  Directorate  of  Evaluation  and  Standardization,  US 
Army  Aviation  Center,  Fort  Rucker,  Alabama.  The  purpose  of  the 
CTE  program  is  to  establish  policies,  procedures  and  responsibili¬ 
ties  for  classroom  evaluations  at  Fort  Rucker.  The  project  in¬ 
cludes  a  comprehensive  plan  that:  (1)  meets  all  DES’ s  classroom 
evaluation  responsibilities,  <2>  is  simple  to  administer,  and  (3) 
is  flexible  enough  to  meet  dynamic  training  requirements. 


HYPOTHESIS 

The  current  DES  CTE  program  is  inadequate  to  meet  all  the 
requirements  outlined  by  TRADOC  regulations  and  USAAVNC  direc¬ 
tives.  By  clearly  defining  all  requirements,  and  using  personal 
expertise  and  other  available  evaluation  materials,  a  more  com¬ 
plete  CTE  program  can  be  developed  to  meet  most  of  DES’s  classroom 
evaluation  responsibilities,  with  currently  available  resources. 


SCOPE 


This  project  is  limited  to  only  developing  a  program  to 
evaluate  classroom  instruction.  The  program  is  designed  to  pro¬ 
vide  a  system  for  subjective  evaluation  of  classroom  instruction 
to  detect  potential  problems  and/or  identify  the  possible  causes 
of  previously  identified  problems.  The  program  is  not  designed  to 
measure  student  performance.  Other  DES  programs  currently  exist 
to  measure  these  changes  in  student  behavior. 


LIMITATIONS 

Time  and  resources  preclude  this  project  from  including  a 
validation  of  the  proposed  CTE  program.  However,  validation  will 
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be  necessary  to  evaluate  the  effectiveness  of  the  proposed  program 
in  meeting  DES’s  classroom  evaluation  responsibilities.  Based 
upon  the  validation  results,  adjustments  may  be  necessary  to  "fine 
tune"  the  program  prior  to  its  final  adoption  by  DES. 


ASSUMPTIONS 

The  development  of  the  Classroom  Training  Evaluation  Program 
was  based  upon  the  following  assumptions: 

1.  TRADOC  requirements  and  USAAVNC  directives  to  DES,  con¬ 
cerning  classroom  evaluation  policies,  will  not  change  in  the  near 
future. 

2.  DES  will  assume  full  responsibility  for  the  training  of 
all  classroom  instructors  and  evaluation  of  all  academic  instruc¬ 
tion  at  Fort  Rucker.  This  responsibility  was  previously  shared 
with  the  Staff  and  Faculty  Department.  This  assumption  is  based 
upon  HQ,  USAAVNC,  initiatives,  in  October  1984,  regarding  the 
reorganisation  of  DES.  (6) 


Chapter  Two 


BACKGROUND 

Background  data  on  the  Directorate  of  Evaluation  and  Standard¬ 
isation,  USAAVNC,  is  necessary  to  develop  a  CTE  program  that  would 
be  suitable  fa*'-  the  organization.  Therefore,  this  chapter  will 
define  the  mission  of  DES,  explain  its  approaches  to  training 
evaluations,  and  describe  the  requirements  for  a  CTE  program. 

This  will  establish  the  basis  for  classroom  evaluations  and  define 
the  parameters  for  a  proposed  CTE  program. 


DES  MISSION 

DES’s  mission,  as  stated  in  USAAVNC  Reg  10-1,  is: 

Represent  the  USAAVNC  as  proponent  agent  for  the  United 
States  Army  Aviation  Standardization  Program  and  act  as  an 
extension  of  the  Office  of  the  Deputy  Chief  of  Staff  for 
Operations  and  Plans,  Department  of  the  Army,  to  monitor 
and  evaluate  Army-wide  implementation  of  the  Aviation  Stan¬ 
dardization  Program?  collect  and  analyze  training  effec¬ 
tiveness  data  as  it  relates  to  unit,  resident,  and  non¬ 
resident  training  programs,  and  provide  feedback  to  the 
training  and  combat  development  processes  (3-19.01). 

This  mission  statement  is  very  general  and  includes  a  number  of 
responsibilities;  however,  it  can  be  distilled  into  two  major 
tasks:  (1)  To  standardize  aviation  related  policies  and  procedures 

throughout  the  US  Army,  and  (2)  Evaluate  the  training  effectiveness 
of  aviation  related  instruction.  The  first  task,  although  very 
important  to  the  US  Army,  does  not  relate  to  classroom  evaluations; 
therefore,  it  is  not  a  part  of  this  study.  However,  the  second 
task,  evaluate  the  training  effectiveness  of  aviation  related  in¬ 
struction,  is  what  drives  the  CTE  program,  and  is  the  focus  of  this 
study.  Within  DES,  the  Evaluation  Division,  is  the  proponent  for 
this  critical  task.  (See  Figure  1  for  DES  Organizational  Struc¬ 
ture.  )  The  next  section  will  discuss  how  DES,  and  more  specifical¬ 
ly  the  Evaluation  Division,  approaches  this  critical  evaluation 
task. 


DES s  TRAINING  EVALUATION  STRATEGY 

The  IJS  Army's  Aviation  Center  is  a  complex  organization  that 
consumes  a  large  portion  of  the  US  Army  training  budget.  It 
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conducts  a  total  of  52  separate  courses  ranging  from  advanced 
flight  training  to  basic  aircraft  maintenance  (2:1-1).  It  is 
responsible  for  all  resident  flight  training  in  the  US  Army  and  is 
the  proponent  for  training  of  all  Army  aircraft  mechanics.  (See 
Figure  1  for  USAAVNC  Organisational  Structure. )  Within  the 
Center,  DES,  through  its  Evaluation  Division,  is  responsible  for 
evaluating  all  these  Programs  of  Instruction  (POI).  Because  the 
Evaluation  Division  is  not  staffed  to  evaluate  all  courses  con¬ 
tinuously,  DES  uses  a  Training  Evaluation  Strategy  that  allows 
them  to  "zero  in"  on  the  most  critical  training  problems. 

The  strategy  uses  a  four — step  procedure  to  manage  the  evalua¬ 
tion  mission  (l:Annex  L) : 

-  Problem  area  detection. 

-  Identif ication/Verif ication  of  the  cause  of  the  problem. 

-  Recommendation  for  change. 

Follow-up. 

The  first  step,  problem  area  detection,  identifies  problems  exist¬ 
ing  with  the  training  product  and/or  in  the  training  environment. 
This  step  provides  a  means  to  consolidate  data  from  all  DES  evalua¬ 
tion  feedback  programs,  so  that  problems  can  be  identified,  priori¬ 
ties  established,  and  investigation  efforts  initiated.  These 
feedback  programs,  operated  by  other  branches  with  DES,  include: 

Branch  training  teams 

-  Instructor  pilot  evaluations 
Graduate  surveys 

-  Student  critiques 

-  Classroom  evaluations 
Test  item  analysis 
Field  performance  checks 

Each  of  these  programs  generate  valuable  information  for  identifi¬ 
cation  of  potential  training  deficiencies.  Individually,  these 
programs  are  less  likely  to  identify  a  problem;  however,  when 
information  from  one  prooram  correlates  information  from  other 
sources,  the  data  become^  more  meaningful  and  valid  in  problem 
detection.  Feedback,  from  these  various  sources,  is  collected 
through  a  monthly  trend  list  that  describes  potential  problems  in 
each  area.  A  program  coordinator  consolidates  these  lists,  and 
convenes  a  quarterly  meeting  to  discuss  each  problem.  Armed  with 
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Figure  1:  UShrvNC  and  DES  Organi sati onal  Structure  for  Training 
and  Training  Evaluation.  (Excludes  all  support  organi zti ons  and 
other  separate  agencies.) 


this  information,  the  director  of  DES  prioritizes  the  potential 
problem  areas  for  more  in-depth  evaluation,  and  establishes 
evaluation  objectives  to  guide  future  information  collection 
efforts  by  the  various  feedback  programs. 

The  second  step,  identification/verification  of  the  cause  of 
the  problem,  is  critical  because  valid  recommendations  for  im¬ 
provement  cannot  be  made  until  the  cause  of  a  problem  is  known. 

This  step  avoids  treating  the  symptoms  of  a  problem,  by  conduct¬ 
ing  in-depth  study  to  identify  the  real  cause (s)  of  training  prob¬ 
lems.  Experience  has  shown  that  in  some  cases  the  real  problem 
may  be  masked.  For  example — an  instructor  experiences  a  steady 
drop  in  student  performance  in  a  particular  block  of  instruction. 
Without  in-depth  study,  it  might  appear  that  the  instructor ' s  per — 
formance  has  fallen  off  and  is  causing  the  problem.  This  may  be 
true;  however,  the  true  cause  could  also  be  that  the  entry  level 
knowledge  of  the  students  had  dropped,  and  that  the  training 
developers  had  not  adjusted  the  instruction  accordingly.  In  this 
case,  replacing  or  attempting  to  correct  the  instructor  would  not 
solve  the  problem.  Although  this  is  a  simplistic  example,  it 
illustrates  the  need  for  in-depth  study  to  identify  the  real  cause 
of  a  problem  before  a  valid  recommendation  for  corrective  action 
can  be  developed.  This  step  normally  requires  additional  study  and 
the  development  of  an  evaluation  plan.  The  evaluation  plan,  in 
most  cases,  will  involve  some  type  of  classroom  evaluation. 

The  third  step,  recommendation  for  change,  is  a  normal  staff 
action.  It  involves  DES  communicating  the  problem  to  the  training 
developer  and  directing  corrective  action.  DES  does  not  develop 
corrective  action.  It  only  identifies  where  the  problem  exists, 
and  then  directs  the  training  departments  to  prepare  corrective 
action. 

The  fourth  step,  follow-up,  is  as  important  as  all  the  other 
step's.  Evaluation  serves  no  useful  function  unless  improvement  in 
the  training  process  occurs.  Therefore,  the  effects  of  a  change  in 
the  training  environment  must  be  evaluated  by  follow-up,  to  detet — 
mine  if  the  training  department  initiated  actions  to  correct  the 
deficiency  and  whether  the  corrective  actions  implemented  were 
effective.  This  step  establishes  a  suspense  system  to  ensure 
effective  follow-up  of  all  recommendations. 


REQUIREMENTS  OF  THE  DES  EVALUATION  STRATEGY 


From  the  preceding  discussion  of  the  DES  Training  Evaluation 
Strategy,  it  is  obvious  that  two  types  of  evaluations  are  required 
to  meet  the  intent  of  the  strategy. 

The  classroom  evaluations  necessary  to  complete  Steps  1  and  4 
are  very  superficial  when  compared  to  the  in-depth  evaluation 


required  to  complete  Step  2.  The  classroom  evaluations  used  to 
complete  Steps  1  and  4  do  not  require  trained  evaluators  since 
they  are  only  surveying  the  training  environment  and  the  actual 
delivery  of  the  training.  However,  Step  2  of  the  DES  Strategy 
requires  an  in-depth  review  of  the  entire  trainining  development 
process  as  detailed  in  TRADOC  Regulation  350-7. 

This  evaluation  of  the  entire  development  process  is  essential 
to  determine  the  actual  cause  of  the  training  problem  identified 
in  Step  1  of  the  strategy.  This  type  of  evaluation  must  address: 

Are  the  correct  tasks  being  taught? 

Are  the  objective  standards  appropriate  to  the  ! ludent  popu¬ 
lation? 

Etc. 

An  evaluation  of  this  nature  will  require  a  skilled  evaluator  that 
is  schooled  in  the  training  envelopment  process.  Therefore,  two 
types  of  evaluation  are  required  to  implement  the  DES  Strategy, 
and  each  type  of  evaluation  requires  a  different  level  of  evalua¬ 
tion  expertise.  The  following  are  the  characteristics  of  the  two 
types  of  evaluation. 

Type  !■  Evaluations. 

Purpose.  Tq  detect  problems  with  classroom  instruction. 
Characteristic^. 

-  Will  involve  the  observation  of  classroom  instruction 

only. 


Evaluators  need  not  to  be  experts  in  training  develop¬ 
ment  but  must  be  familiar  with  accepted  classroom  instructional 
procedures  at  U5AAVNC. 

Will  fulfill  the  requirements  for  Steps  1  and  4  of 
DES’s  Training  Evaluation  Strategy. 

Type  2  Evaluations. 

Purpose.  To  determine  if  classroom  instruction  contributes 
to  a  particular  problem,  and  if  so,  to  identify  what  is  wrong  with 
thr  instruction. 

Characteristi cs. 

-  Evaluators  must  be  well  versed  in  the  training  develop¬ 
ment  process. 
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Evaluators  must  be  trained  to  conduct  training  evalua¬ 
tions. 

-  Will  require  an  analysis  of  the  training  development 
process,  as  well  as,  classroom  instruction  observations. 

-  Will  -fulfill  requirements  for  step  2  of  DES’s  Training 
Evaluation  Strategy. 

This  interpretation  of  the  two  types  of  evaluations  is  based 
upon  established  requirements.  Although,  all  the  requirements 
mentioned  do  not  necessitate  highly  trained  specialist,  by  cate¬ 
gorising  classroom  evaluations  into  these  two  distinct  types,  DES 
gains  greater  flexibility  in  tailoring  the  available  evaluation 
resources  to  a  particular  situation. 


Chapter  Three 


CURRENT  ENVIRONMENT 

This  chapter  will  describe  the  currect  DES  classroom  evaluation 
program  and  highlight  its  i nadequaci es,  outline  the  resources  DES 
is  willing  to  commit  to  a  CTE  program,  and  discuss  the  existing 
evaluation  material  that  was  helpful  in  developing  a  new  CTE  pro¬ 
gram  -for  Fort  Rucker.  In-formation  provided  in  this  chapter, 
coupled  with  the  requirements  -for  a  CTE  program  established  in 
Chapter  2,  provide  the  basis  -for  the  program  outlined  in  Chapter  4. 


CURRENT  DES  CLASSROOM  EVALUATION  PROGRAM 

The  current  USAAVNC  Classroom  Evaluation  Program  is  conducted 
by  DES  in  conjunction  with  the  Sta-f-f  and  Faculty  Department,  Direc¬ 
torate  of  Training  and  Doctrine.  The  Staff  and  Faculty  Department 
is  involved  in  evaluation  because  they  are  currently  responsible 
for  the  training  of  all  academic  instructors.  As  indicated  in 
Chapter  1,  Assumption  Number  2,  this  function  will  revert  to  full 
DFS  responsibility  in  the  near  future.  Therefore,  this  project 
addressed  DES  as  the  only  responsible  evaluation  agency. 

The  Internal  Evaluation  Branch,  within  the  Evaluation  Division, 
is  responsible  for  the  classroom  evaluation  program  in  DES.  Cur - 
renU>,  there  are  no  procedures  that  cleariv  outline  the  intent  o'i 
this  program  or  delineate  responsibilities.  The  program  is 
basically  accomplished  through  verbal  guidance  and  procedures,  and 
relies  on  the  use  of  an  observation  checklist.  Evaluators,  from 
the  Staff  and  Faculty  Department  and  DES,  attend  32  randomly 
selected  classes  each  week.  These  evaluators  complete  the  observa¬ 
tion  ihecHist  at  the  end  of  each  evaluation,  and  forward  the  form, 
through  command  channels,  to  the  Assistant  School  Commandant. 

This  ongoing  program  fulfills  the  requirements  of  higher 
headquarters  and  marginally  meets  the  needs  of  Step  1  in  DES's 
Training  Evaluation  Strategy.  The  checklist  used  in  this  eval  ua- 
t i on  provides  a  cursory  look  at  the  instruction,  and  with  minimum 
' hangea  it  could  meet  the  needs  of  a  Type  1  Evaluation.  However, 
formal  procedures  must  be  established  to  determine:  which  classes 
f-r  e  reviewed,  how  evaluators  are  selected,  and  how  to  ensure  re- 
s-'-l  \  *re  tracked  and  incorporated  into  the  Training  Evaluation 
g-,  . 


? 


The  current  classroom  evaluation  program  does  not  fulfill  the 
requirements  established  for  Type  2  Evaluations.  There  are  also  no 
procedures  to  determine  "why"  the  classroom  instruction  is  not 
meeting  its  objective,  i *  the  problem  is  not  readily  apparent  in 
the  classroom.  As  mentioned  in  Chapter  2,  this  function  is  criti¬ 
cal  f«-*r  DE:8  to  make  valid  recommendations  to  the  training 
devel oper . 


AVAILABLE  RESOURCES 

The  Internal  Evaluation  Branch  currently  has  two  civilians  and 
three  military  evaluators  capable  o^  conducting  classroom  evalua¬ 
tions.  These  individuals  are  also  responsible  for  a  number  of 
other  duties  and  represent  the  only  personnel  available  for  Type  2 
Evaluations.  DES,  as  a  whole,  has  a  nunber  of  other  officers  who 
can  effectively  perform  Type  1  Evaluations.  These  individual's 
periodically  at vend  classes  for  a  variety  of  reasons  associated 
with  their  individual  responsibilities,  and  if  properly  briefed, 
could  be  used  to  conduct  Type  t  Evaluations.  However,  the  total 
resources  available  for  the  CTE  program  are  really  dependent  upon 
command  influence  and  priorities.  Therefore,  DES  and  the  Evalua¬ 
tion  Division  leadership  must  be  the  ones  t.o  establish  the  re¬ 
sources  available  for  the  program  based  upon  the  director's  priori¬ 
ties.  In  an  interview  conducted  on  20  Qctober  1984,  the  Deputy 
Director  of  DES  and  the  Internal  Evaluation  Branch  Chief  provided 
the  following  guidance  concerning  available  resources  for  the  CTE 
program: 

,  \a)  Current  classroom  evaluations  require  approximately 
four  to  five  man-hours  per  evaluation.  Type  1  Evaluations  should 
be  structured  similarly. 

<b>  Plan  no  more  than  12  Type  1  Evaluations  per  week.  More 
can  be  conducted  if  resources  from  other  DES  branches  are  avail¬ 
able. 


(c)  Do  not  program  any  specific  number  of  Type  2  Evaluations. 
Their  frequency  will  be  determined  through  the  Training  Evaluation 
Strategy  process.  This  will  allow  the  organi nation  to  plan  its  re¬ 
source  allocation  based  on  the  priorities  of  the  identified  problem 
areas. 


•. d)  Manpower  demands  for  a  Type  2  Evaluation  are  situation 
dependent,  but  will  normally  require  in  excess  of  20  man-hours. 

This  requirement  must  be  recognised  when  establishing  priorities 
for  Type  2  Evaluations,  and  does  not  need  to  be  addressed  in  the 
CTE  program  itself. 

in)  The  mosl  important  goal  for  the  CTE  program  is  to  develop 
policies  a i id  pr ocpdures  for  classroom  evaluations.  These  proce- 
dure1.-  must  be  flexible  to  allow  for  the  frequent  change  that  can  be 
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anticipated  in  the  evaluation  schedules.  The  exact  number  of 
evaluations  conducted  each  week  will  vary  and  will  be  determined  by 
the  Internal  Evaluation  Branch  Chief  based  upon  available  manpower 
resources. 


EXISTING  EVALUATION  MATERIAL 


Research  of  available  evaluation  material  was  important  in 
developing  a  new  CTE  program.  In  fact,  the  Instructional.  Systems 
Development  Process  requires  a  product  search  to  determine  if 
materials  exist  which  meet  the  stated  requirement,  and  are  aval  1-- 
able  and  appropriate  for  use  in  the  program  being  developed.  This 
process  prevents  "reinventing  the  wheel",  saves  time  and  re¬ 
sources,  and  provides  a  proven  and  tested  product.  The  develop¬ 
ment  of  the  Type  1  and  2  Evaluation  instruments  was  no  different. 
All  Army  training  installations  are  organized  similarly  with  each 
having  a  DES  with  a  mission  similar  to  DES  at  Fort  Rucker.  How¬ 

ever,  how  they  accomplish  this  mission  varies  between  installa¬ 
tions.  Many  have  viable  classroom  evaluation  programs  that  could 
be  of  some  use  at  Fort  Rucker,  Civilian  sources  are  also  avail¬ 
able  and  could  contribute  to  the  development  of  this  evaluation 
program.  This  section  will  discuss  the  sources  which  were  useful 
in  developing  the  proposed  classroom  evaluation  program  outlined 
in  the  next  chapter. 

TRADOC  Schools. 

<a>  Research  of  other  TRADOC  schools  produced  a  number  of 
checklists  that  were  useful  in  developing  the  Type  1  CTE  checklist 
for  Fort  Rucker.  There  were  no  checklists  which  could  be  imple¬ 
mented  "as  is";  however,  checklists  used  at  the  US  Army  Chemical 
School  and  the  US  Army  Intelligence  School  were  very  similar  to 
what  will  be  required  at  Fort  Rucker.  Portions  of  these  checklists 
were  used  in  the  development  of  the  Type  1  Evaluation  checklist. 

(b>  There  were  no  CTE  programs  which  would  be  useful  for  the 
Type  2  Evaluations.  However,  information  received  from  the  TRADOC 
Evaluation  Team  was  useful  in  determining  the  items  that  are  re- 
qui  red  for  Type  2  Evaluations. 

TRADOC  Rea  550-7 — A  Systems  Approach  to  Training.  Th i s  r egul at i on 
provides  general  information  on  what  a  school  evaluation  program 
should  include.  It  does  not  provide  specific  guidance  or  informa¬ 
tion  on  classroom  evaluations  but  does  provide  some  general  guide¬ 
lines,  The  regulation  stipulates  the  minimum  requirements  for  the 
training  development  process,  and  these  requirements  were  very 
useful  in  determining  the  appropriate  items  to  include  in  the  Type 
Evaluation. 

Civilian  Sources.  Research  of  civilian  sources  did  not  produce  any 
specific  information  applicable  to  Fort  Rucker.  Ample  resources 


11 


were  available  an  evaluation  techniques  and  methods  tor  the 
measurement  ot  student  performance,  and  this  provided  useful  back¬ 
ground  information;  however,  there  was  little  information  on  a 
means  of  curriculum  evaluation  that  was  applicable  to  IJSAAVNC. 

(The  sources  used  for  general  background  information  are  included 
3  ri  the  bibliography.) 


NOTE 


Chapter  4  of  this  project  is  designed  to  "stand  alone"  as  a  program 
directive  for  the  Directorate  of  Evaluation  and  Standardisation,  US 
Army  Aviation  Center,  Fort  Rucker,  Alabama. 
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Chapter  Four 


DES  CLASSROOM  TRAINING  EVALUATION  PROGRAM 

I.  INTRODUCTION. 


1 .  Ref erences. 

a.  TRADOC  Reg  350-7  (Draft):  A  Systems  Approach  to 
Training. 

b.  USAAVNC  Reg  310-4:  Preparation  and  Use  of  Lesson 
Plans  and  Instructor  Guides. 

c.  USAAVNC  Reg  350-4:  Classroom  Management. 

d.  DES  Policy  and  Procedure  Guide. 

e.  Internal  Evaluation  Branch  SOP. 

2.  General :  Classroom  training  evaluations  are  a  fundamental 
component  of  DES's  Training  Evaluation  Strategy.  They  are  used  to 
gather  information  concerning  the  quality  of  instruction  as  it 
occurs  and/or  to  clarify  the  nature  and  cause  of  problem  areas 
previously  identified.  As  outlined  in  Annex  L,  of  DES’s  Policy  and 
Procedure  Guide,  the  Classroom  Training  Evaluation  (CTE)  program  is 
the  responsibility  of  the  Evaluation  Division,  and  is  managed  by 
the  Internal  Instructional  System  Evaluation  Branch  (IEB).  How¬ 
ever,  all  personnel  involved  with  evaluations  must  fully  support 
the  program  for  it  to  achieve  its  goal  of  TRAINING  IMPROVEMENT. 

3.  Purpose:  This  program  establishes  policies  and  procedures 
for  the  Evaluation  Division,  DES, to  manage  its  classroom  training 
evaluation  responsibilities.  It  provides  a  process  to  plan,  con¬ 
duct,  consolidate  and  report  classroom  evaluations. 

4.  Scope:  The  program  contains  a  standardized  approach  to 
classroom  evaluations.  It  is  not  intended  to  cover  every  possible 
classroom  evaluation  requirement;  rather,  it  is  designed  to  provide 
guidelines  that  meet  most  recurring  needs.  The  program  is  also 
not  designed  to  evaluate  flight  line  instruction;  although,  some 
portions  of  the  program  may  be  useful  in  this  instructional  area. 

5.  Background:  The  program  was  developed,  during  the  first 
quarter  of  FY  1985,  to  satisfy  the  Staff  Problem  Solving  Project 
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requirement  necessary  tor  completion  of  the  Air  Command  and  Statt 
College.  It  is  based  upon  an  analysis  of  DES’s  classroom  evalua¬ 
tion  requirements  and  available  resources.  The  type  of  classroom 
evaluations,  established  in  this  study,  were  driven  by  the  require 
ments  outlined  in  DES's  Training  Evaluation  Strategy  and  directives 
from  higher  headquarters.  Other  TRADOC  installations  were  contac¬ 
ted  in  an  effort  to  find  material  that  would  be  useful  in  the 
development  of  a  program  for  Fort  Rucker.  Although  a  complete 
program  was  not  available,  useful  information  was  received  from  the 
Army  Intelligence  School,  Army  Chemical  School,  the  Army  Training 
Board  and  the  TRADOC  School  Evaluation  Team.  The  checklists 
developed  during  the  study  incorporate  selected  portions  of  the 
material  received  from  these  sources. 

II.  PROCEDURES 

The  CTE  program  is  built  around  the  use  of  two  distinct  types 
of  classroom  evaluations,  that  will  meet  most  of  the  DES  classroom 
evaluation  requirements.  This  section  provides  an  explanation  of 
the  prescribed  procedures  for  each  type  of  classroom  evaluation. 

1.  Type  1  -  Classroom  Training  Evaluations. 

a.  Purpose.  A  Type  1  CTE  is  used  to  detect  problems  in  • 
classroom  instruction.  It  is  not  intended  to  be  an  in-depth  analy¬ 
sis  of  a  particular  course  of  instruction.  It  is  intended  to  be  an 
unannounced  inspection  of  training,  that  will  require  limited  pre¬ 
paration  time,  and  can  be  conducted  by  most  officers  assigned  to 
DES.  Type  1  CTEs  are  designed  tos 

<1)  Provide  a  snapshot  of  training,  to  be  compared 
with  other  feedback  sources  to  identify  problem  areas  in  the  train¬ 
ing  system. 

(2)  Assist  in  standardizing  classroom  instruction  at 

USAAVNC. 


<3>  Ensure  training  departments  take  appropriate 
action  on  previously  submitted  recommendations  for  improvement. 

b.  Selection  of  Classes  for  Observation.  The  CTE  Pro¬ 
gram  Coordinator  (PC) , appointed  by  the  IEB  commander,  selects  the 
classes  to  be  evaluated.  Selections  should  comply  with  the  follow¬ 
ing  general  guidelines: 

(1)  A  minimum  of  12  evaluations  per  week  must  be 

accomplished. 

(2)  Approximately  50  percent  of  the  evaluations 
should  be  selected  at  random,  and  over  an  extended  period  be  equal¬ 
ly  divided  between  the  training  departments.  This  will  assist  in 


standardising  instruction  at  USAAVNC  and  ensure  no  instructional 
area  is  overlooked. 


(3)  The  remaining  50  percent  of  the  evaluations 
should  be  targeted  in  selected  areas  that  have  been  identified 
potential  problem  areas.  This  will  increase  the  effectiveness  of 
the  program  by  concentrating  in  areas  that  will  produce  the  great¬ 
est  dividends.  Some  sources  available  to  assist  the  Program  Coot — 
dinator  in  establishing  these  targeted  areas  are: 


-  Higher  headquarters  direction 
Student  critiques 

-  Test  item  analysis 

Areas  of  concern  established  dur-ing  the  last 
Quarterly  Training  Evaluation  Strategy  session. 

Follow-up  evaluations  of  previous  recommenda¬ 
tions. 


These  guidelines  are  not  intended  to  "tie  the  hands"  of  the  PC 
but  to  assist  him  in  establishing  a  creditable  program.  The  PC 
must  develop  a  system  to  manage  the  evaluation  schedule.  The 
system  should  be  one  that  meets  the  PC’s  individual  style,  but  is 
easily  understood  by  other  members  of  the  IEB.  This  system  must 
ensure  that:  general • guidel ines  are  followed,  the  selected  target 
areas  are  appropriate,  and  evaluators  are  provided  adequate  pre¬ 
paration  time  (at  least  one  week  advance  notification).  The  PC 
must  remain  flexible  and  adjust  his  evaluation  schedule  to  meet 
changing  requirements. 

c.  Selection  of  Evaluators. 

The  PC,  in  conjunction  with  the  IEB  commander,  assigns 
evaluators  to  the  classes  selected  for  evaluation.  Since  indi¬ 
viduals  in  the  Evaluation  Division  are  assigned  course  areas  of 
responsibility,  evaluator  assignments  should  coincide  with  these 
areas  if  possible.  When  an  individual,  selected  as  evaluator,  is 
not  assigned  to  the  IEB,  the  PC  must  coordinate  with  the 
individual’s  supervisor  before  assigning  him  evaluator  duties. 

The  structure  of  the  Type  I  CTE  does  not  require  a  highly 
qualified  educational  specialist.  It  only  requires  an  individual 
familiar  with  Army  training,  and  when  properly  briefed,  capable  of 
observing  training  and  recording  the  results.  Most  officers  and 
educational  specialists  assigned  to  DES  are  capable  of  conducting 
a  Type  I  CTE.  However,  in  some  cases,  an  individual  with  specific 
qualifications  may  be  required  (e.g.,  requirement  to  evaluate 
technical  information  on  an  aircraft  may  require  an  SIP).  The  PC 
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and  IEB  commander  must  keep  this  requirement  in  mind  when  assign¬ 
ing  individuals  to  the  evaluation  schedule. 

The  PC  will  ensure  all  evaluators  are  properly  brie-fed 
prior  to  their  performance  of  any  evaluation.  This  does  not  mean 
that  the  PC  must  brief  every  evaluator  before  every  evaluation.  It 
implies  that  the  PC  must  ensure  the  evaluator  is  familiar  with  the 
program  and  understands  the  requirements  outlined  in  the  following 
section. 


d.  Conduct  of  Evaluation  (Evaluator  Actions/Duties). 
All  evaluators  will  use  the  following  procedures  when  conducting 
Type  I  CTEs. 


( 1 )  Before  the  Evaluation. 


<a)  Contact  the  PC  and  ensure  you  understand  the 
purpose  of  the  evaluation. 

(b>  Review  the  IEB  working  file.  Look  for  com¬ 
pleted  or  ongoing  actions  that  might  effect  the  outcome  of  your 
evaluation. 


(c)  Obtain  a  current  Program  of  Instruction  (POD 
that  is  on  file  in  the  IEB  library,  and  review  the  block  dealing 
with  this  evaluation.  Reproduce  a  copy  of  the  block  of  instruction 
you  are  scheduled  to  evaluate.  Use  this  copy  to  compare  the  POI 
entry  with  the  lesson  plan  and  student  handouts. 

<d)  Obtain  a  training  schedule  and  ensure  train¬ 
ing  is  scheduled  appropri atel y.  (Training  schedules  are  main¬ 
tained  by  the  IEB). 

(e)  Pick  up  your  evaluation  packet  from  the  PC 
and  ensure  you  are  familiar  with  its  contents.  If  you  have  ques¬ 
tions,  ask  the  PC.  The  packets  should  contains 


is  required) 


Training  observation  forms  (a  minimum  of  3 


-  USAAVNC  PAM  310-4;  Preparation  and  use  of 
Lesson  Plans  and  Instructor ? s  Guide 


USAAVNC  PAM  350-4;  Classroom  Management 
(2)  During  the  Evaluation. 

(a)  Arrive  at  the  training  site  NLT  10  minutes 
before  the  training  is  scheduled  to  begin.  Introduce  yourself  to 
the  Principal  Instructor  (PI),  and  state  why  you  l'i~e  there.  Ask  to 
be  seated  at  or  near  the  seat  reserved  for  visitors.  It  is 
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important  that  you  locate  yourself  where  you  can  see  all  the  training 
without  interfering  with  the  training  or  expected  visitors. 

(b)  Inspect  the  visitor’s  -folder.  It  should  be 
organised  and  contain  the  information  outlined  in  USAAVNC  PAM  350- 
4,  If  there  is  no  Lesson  Plan  <LP) ,  ask  the  PI  for  a  copy  so  you 
can  complete  your  worksheet  and  follow  the  training.  You  also  need 
to  follow  the  instruction  with  the  student  handout (s). 

(c)  Review  the  training  and  complete  the  obser — 
vation  form  (minimum  of  three  copies  required) .  These  observations 
can  lead  to  changes  in  the  training  program;  therefore,  be 
thorough,  accurate,  and  objective  in  reporting  your  observations. 

Keep  in  mind  that  this  is  an  evaluation  of  the  training  program, 
not  just  the  instructor.  Consider  each  item  on  the  worksheet 
separately  and  record  only  what  actually  occurs.  Only  record  items 
that  can  be  easily  supported  by  a  documented  reference.  When  a 
problem  is  recognised,  do  not  hesitate  to  record  it,  but  ensure 
good,  as  well  as,  bad  comments  are  recorded. 

(d)  Do  not  interfere  with  the  training.  Try  to 
be  inconspicuous  when  taking  notes,  since  this  can  be  an  annoying 
distraction  in  the  classroom  environment. 

(3)  After  the  Evaluation. 


(a)  Return  all  training  materials  to  the 
Visitor’s  Folder  or  the  PI.  Leave  your  area  as  you  found  it! 
Respect  the  fact  that  the  training  area  belongs  to  the  PI. 

<b>  Give  one  copy  of  the  completed  worksheet  to 
the  PI  and  thoroughly  brief  him  on  your  observations.  Avoid 
arguments  over  controversi al  or  unclear  content  of  the  training. 
The  best  way  to  avoid  this  situation  is  to  be  accurate  in  your 
evaluation  and  know  that  there  is  a  documented  reference  to  sup¬ 
port  your  comments. 

(c)  Return  the  original  and  remaining  copy  of 
the  observation  form,  along  with  the  evaluation  packet,  to  the  PC. 
Highlight  any  information  that  you  feel  deserves  immediate  atten¬ 
tion,  and  if  necessary,  personally  debrief  the  PC  on  your  find¬ 
ings. 


e.  Post  Evaluation  Action.  No  individual  report,  other 
than  the  completed  observation  form,  is  required  for  a  Type  I  CTE. 
The  PC  collects  all  Type  I  Evaluations  and  is  responsible  for 
maintaining  a  CTE  program  log  and  a  file  of  all  observation  forms. 
The  CTE  program  log  will  include:  the  block  of  instruction 
observed,  responsible  training  department,  when  the  evaluation  was 
conducted,  evaluator’s  name,  instructor’s  name,  and  major  findings 
(if  any).  If  follow-up  action  is  required,  the  PC  will  annotate 
this  fact  in  the  log  and  establish  a  suspense  system  to  ensure 
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follow-up  action  is  completed.  This  log  also  serves  as  a  useful 
guide  when  scheduling  classes  -for  evaluation. 

Weekly,  the  F'C  will  send  a  copy  of  each  observation  re¬ 
port,  through  channels,  to  the  Assistant  School  Commandant  for 
review.  The  reports  should  be  sent  "as  is";  however,  the  PC  must 
ensure  they  are  legible  and  easily  understood. 

Each  month,  the  PC  also  develops  a  list  of  potential 
problem  trends  identified  through  the  CTE  program,  and  provides  it 
to  the  IEB  commander  as  required  by  DES’s  Training  Evaluation 
Strategy.  The  CTE  trend  list  is  compared  to  data  provided  by  other 
feedback  sources,  which  may  clarify  problems  or  identify  major 
problems  areas  requiring  further  evaluation;  therefore,  the  trend 
list  should  include  all  potential  problems  identified  through  the 
CTE  program. 

2.  Type  2  -  Classroom  Training  Evaluations. 

a.  Purpose.  The  purpose  of  a  Type  2  dTE  is  to  determine 
if  classroom  instruction  contributes  to  a  previously  identified 
problem  and  to  identify  what  is  wrong  with  the  instruction.  It  is 
designed  to  identify  why  a  particular  problem  exists,  and  will 
require  an  in-depth  analysis  of  the  block  of  instruction.  It  is 
only  conducted  if  an  identified  training  problem  requires  additional 
evaluation  to  pinpoint  the  exact  cause/source  of  the  problem.  The 
Type  2  CTE  will  also  fulfill  another  DES  responsibility:  evalua¬ 
tion  of  the  training  development  process.  It  provides  a  mechanism 
for  ensuring  the  training  agencies  at  USAAVNC  are  complying  with 
the  Systems  Approach  to  Training  process.  This  requirement,  al¬ 
though  not  part  of  the  CTE  program,  is  an  important  role  for  DES, 
arid  can  be  fulfilled  with  a  Type  2  CTE. 

b.  General  Characteri sties.  ■  A  Type  2  CTE  will  normally 
require  a  study  plan  and  will  always  require  a  final  report.  It 
Ls  a  thorough  review  of  the  training  development  process  for  a 
particular  block  of  instruction,  followed  by  a  Type  1  CTE.  It  is 
intended  to  determine  if  the  instruct! onal  design  is  appropriate, 
and  in  turn,  if  the  instruction  is  being  conducted  as  designed. 

In  so  doing,  it  will  provide  data  to  determine  the  cause  of  an 
identified  training  problem. 

A  Type  2  CTE  is  not  intended  to  be  a  major  study.  The 
conduct  of  major  studies  are  outlined  in  the  IEB  SOP.  A  Type  2 
CTE  is  designed  to  clearly  define  a  problem  in  a  particular  block 
of  instruction,  and  trace  it  through  the  curriculum  development 
process,  to  the  classroom  instruction,  to  determine  its  cause. 

The  evaluation  may  be  terminated  when  the  evaluator  has  identified 
the  nature  of  the  problem.  A  Type  2  CTE  may  be  very  short  if  the 
problem  is  discovered  quickly;  however,  if  the  investigation  must 
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go  through  the  complete  Instructi onal  Systems  Development  Process 
it  will  require  significant  time  and  effort. 

c.  Scheduling.  Type  2  CTEs  are  managed  by  the  GTE 
program  coordinator  and  scheduled  through  DES's  quarterly  Training 
Evaluation  Strategy  meeting.  Scheduling  a  Type  2  GTE  is  based 
upon  the  resources  available,  the  urgency  of  need,  and  priorities 
of  other  evaluation  requi rements.  Type  2  CTEs  may  also  be  re¬ 
quired  as  part  of  a  major  study.  In  this  case,  resources  for  the 
TTE  are  included  in  the  study's  plan. 

d.  Selection  of  Evaluators.  Evaluators  for  Type  2  CTEs 
are  selected  by  the  IEB  commander.  They  must  be  trained  instruc¬ 
tional  system  evaluators,  familiar  with  the  training  development 
pr  nre&s  and  classroom  instruction  techniques.  Therefore,  only 
personnel  assigned  to  the  Evaluation  Division,  and  primarily  the 
Internal  Evaluation  Branch,  should  be  selected  as  evaluators.  As 
with  Type  t  CTEs,  evaluators  should,  when  passible,  be  selected  in 
their  responsible  course  area(s). 

e.  Procedures.  The  procedures  tor  a  Type  2  GTE  are 
intended  to  be  very  flexible  to  allow  the  evaluator  maximum  freedom 
of  action.  Unlike  Type  1  CTEs,  evaluators  are  trained  instruc¬ 
tional  system  evaluators,  and  should  not  require  much  guidance. 
Therefore,  Type  2  CTEs  will  only  require  an  informal  study  plan. 

The  study  plan  need  not  be  as  detailed  as  required  for  a  major 
study  and  does  not  need  to  be  published.  However,  it  must  be 
approved  by  the  PC  and  IEB  Commander  to  ensure  it  addresses  the 
targeted  problem  area.  As  a  minimum,  the  study  plan  must  outline 
the  purpose  of-  the  study  and  establish  Essential  Elements  of 
Analysis  (EFA) . 

EEAs  should  be  established  for  each  phase  of  the  Systems 
Approach  to  Training  (Analysis,  Design,  Develop,  Implement)  and 
will  depend  upon  the  nature  of  the  problem  being  studied.  The  EEAs 
should  be  researched  in  the  order  established  for  the  development 
process,  so  that  once  the  evaluator  is  assured  of  the  nature  of  the 
problem,  he  can  terminate  the  project.  It  is  necessary  to  conduct 
research  in  the  proper  order,  because  it  is  difficult  to  determine 
the  effectiveness  of  course  instruction  or  course  design  without 
first  understanding  the  tasks  established  for  instruction.  Sample 
EEAs  for  each  phase  of  the  Systems  Approach  to  Training  are  at 
F i gure  3. 

A  final  report  is  always  required  for  a  Type  2  Evaluation. 
It  is  not  as  extensive  as  a  major  study  but  must  include  the 
following  items:  (Definitions  are  the  same  as  those  established  in 

Annex  F,  IEF  SOP.) 

Purpose 


Bac kground 


Essential  Elements  of  Analysis 


-  Results 
Conclusion 
Recommendation 

The  report  does  not  need  to  be  a  published  document; 
therefore,  it  should  be  completed  as  a  routine  DF  and  addressed  to 
the  appropriate  training  department.  Its  intent  is  to  quickly 
convey  the  results  of  the  evaluation  and  make  appropriate  recommen¬ 
dations  to  the  training  developers,  so  that  corrective  action  can 
be  quickly  implemented. 


II I .  RESPONSIBILITIES 


The  overall  responsibility  for  the  CTE  program  rests  with  the 
Evaluation  Division  Commander.  Specific  responsibilities  for  in¬ 
dividuals  involved  in  the  program  are  outlined  below: 

t .  Internal  Evaluation  Branch  Commander. 

a.  Supervise  and  monitor  the  CTE  program  for  the 
Evaluation  Division. 

b.  Appoint  a  CTE  Program  Coordinator. 

c.  Designate  Type  1  and  Type  2  CTE  evaluators.  (The  PC 
assists  the  IEB  commander  in  this  function.) 

d.  Consolidate  CTE  feedback  into  the  DES  Training 
Evaluation  Strategy. 

2 .  CTE  Program  Coordi nator (PC) . 

a.  Manage  the  CTE  Program. 

b.  Select  classes  to  receive  Type  1  CTEs. 

c.  Schedule  Type  1  and  Type  2  CTEs. 

d.  Assist  the  IEB  commander  in  designating  Type  1  and 
Type  2  CTE  evaluators. 

e.  Coordinate  with  supervisors  for  the  use  of  evaluators 
assigned  outside  the  IEB  branch. 

f.  Brief  Type  1  CTE  evaluators,  and  ensure  established 
evaluation  procedures  are  followed. 
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g.  Maintain  a  CTE  Program  Log  and  a  -file  copy  of  all 
Type  1  CTEs. 

h.  NLT  Friday  of  each  week,  forward  a  copy  of  all  Type  1 
CTEs,  conducted  during  the  week,  to  the  Assistant  School  Comman¬ 
dant. 

i.  NLT  the  first  week  of  each  month,  analyze  the  results 
of  all  Type  1  CTEs,  and  submit  a  CTE  trend  list  to  the  IEB 
commander . 

j.  Monitor  the  conduct  of  Type  2  CTEs. 

k.  Develop  a  CTE  program  suspense  system  to  ensure  the 
timely  follow-up  of  all  recommendations. 

3.  Evaluators. 

a.  Plan,  conduct,  and  report  assigned  CTEs  as  outlined  in 
this  program. 

b.  Coordinate  with  the  CTE  PC  for  schedule  changes  or 
any  assistance. 

c.  Recommend  changes  to  this  program  as  appropriate. 


CLASSROOM  TRAINING  EVALUATION 
OBSERVATION  CHECKLIST 


Block  of  Instruction: 

Sub  iect: 

Course 

Instructor:  Name 

Unit 

Phone 

Evaluator:  Name 

Unit 

Phone 

Evaluation  Location 

Date 

Time:  Start 

Stop 

_ 

A.  TRAINING  ENVIRONMENT  AND  PERSONAL  CONDUCT. 

1.  Training  area:  set  up  as  outlined  in  lesson 
plan  and  complies  with  USAAVNC  REG  350-4. 

2.  Learning  environment:  conducive  to 
learning.  (e.g.;  lighting,  noise, 
temperature,  distractions) . 

3.  Personal  conduct  and  appearance:  IAW  USAAVNC 
REG  350-4. 

4.  Visitors  Folders:  available  and  IAW  USAAVNC 
REG  350-4. 

COMMENTS  _ 


YES  NO  N/A 


B.  TRAINING  MATERIALS 

5.  Lesson  plan  or  instructor’ s  guide:  IAW  USAAVNC 
PAM  310-4;  meets  intent  of  instruction 
outlined  in  POI  scope. 

6.  Student  handouts:  follows  instructional  flow 
and  lesson  plan  design;  easily  understood; 
accurate  (technically  and  administrati vely 

7.  Training  aids  and  equipment:  on  hand 

and  functional;  appropriate  for  instruction. 

Comments  _ 


Figure  2:  Type  1  CTE  Observation  Checklist 


23 


C.  CONDUCT  OF  TRAINING 


YES  NO  N/A 

8.  Training  Time:  conducted  IAW  schedule  and 

lesson  plan;  too  much  or  too  little.  _  _  _ 

9.  Training  Objectives:  clearly  stated; 

outlined  in  lesson  plan  and  student 
handout;  meets  intent  of  training  scope 

in  POI.  _  _  _ 

10.  Instructor  Knowledge:  demonstrates  competance 

as  SME;  technically  accurate;  well 

informed.  _  _  _ 

11.  Instructor  Delivery:  well  rehearsed;  good 

platform  techniques  <e.g.;  articulate, 
understandable,  good  eye  contact,  not 

monotone) .  _  _  _ 

12.  Classroom  management:  student  questions 

solicited  and  answered;  discussion 

relevant;  IAW  USAAVNC  REG  350-4.  _  _  _ 

COMMENTS  _ _ 


OVERALL  ASSESSMENT:  Check  One 


Outstanding  _ ;  Above  Average  _ ;  Average  _ ; 

Below  Average  _ ;  Unsatisfactory  _ 

COMMENTS  (Required  for  below  average  and  unsati sf actory  ratings) 


Observer’s  signature 


(This  Form  Must  Be  Completed  in  Triplicate) 
Figure  2:  Continued 
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Sample  EEAs  -for  each  phase  of  the  system's  approach  to  training  are 
listed  below.  The  list  only  provides  a  representati ve  sample  o-f 
the  type  EEAs  that  should  be  established  during  a  Type  2  CTE. 

ANALYSIS  EEAs;  In-formation  available  through  course  Development 
Division,  DOTD. 

1.  Are  there  critical  tasks  selected  for  training  that 
cover  the  problem  area  being  researched?  (Many 
times  the  task  was  not  selected  for  training  at  the 
training  base.) 

2.  Is  the  instructional  site  appropriate? 

3.  Are  tasks  performace  measures  (TPM)  adequate? 

4.  Have  TPMs  been  verified? 

5.  Is  the  audit  trail  accurate  and  up  to  date? 

6.  Is  the  target  population  description  accurate? 

DESIGN  EEAs:  Information  available  through  responsible  training 
department. 

1.  Are  the  terminal  learning  objectives  (TLD)  written 
I AW  appropriate  directives? 

2.  Do  the  TLOs  reflect  the  TPMs? 

3.  Are  the  TLOs  achievable  by  the  target  population? 

4.  Is  the  training  sequence  conducive  to  effective 
learning? 

5.  Can  the  learning  objectives  be  measured? 

DEVELOP  EEAs:  Information  available  through  responsible  training 
department. 

1.  Is  the  training  management  plan  adequate? 

2.  Are  the  lesson  plans  written  IAW  USAAVNC  PAM  310-4? 

3.  Does  an  audit  trail  of  actions  exist,  and  is  the  audit 
trail  accurate  and  up  to  date? 

4.  Is  instructor  training  adequate  to  support  the 
block  of  instruction? 


Figure  3:  Type  2  Sample  Essential  Elements  of  Analysis 
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IMPLEMENT  EEAs:  In-formation  available  through  the  responsible 
training  department. 

1.  Is  the  training  conducted  as  planned? 

2.  Are  students  meeting  the  learning  objectives? 

3.  Are  training  data  and  information  collected  in 
support  of  validation  and  evaluation  efforts? 

4.  Is  the  training  conducted  IAW  USAAVNC  directives? 

•  5.  Are  the  instructors  properly  trained  to  teach  the 

material? 


Chapter  Five 


CONCLUSIONS  AND  RECOMMENDATIONS  v 

This  study  has  established  the  need  for  a  Classroom  Training 
Evaluation  Program,  examined  the  requirements  for  the  program,  and 
developed  a  workable  CTE  program  for  DES.  This  chapter  will  focus 
on  conclusions  and  recommendations  concerning  the  project. 


CONCLUSIONS 


Several  conclusions  can  be  drawn  from  this  study. 

1.  The  current  CTE  program  at  USAAVNC  is  inadequate  to  meet 
all  DES  classroom  evaluation  requirements.  The  current  program  is 
very  superficial,  does  not  contain  formalized  responsibilities  and 
procedures,  and  only  meets  the  needs  of  USAAVNC  Headquarter ” s 
directives.  It  also  does  not  provide  adequate  input  into  the  DES 
Training  Evaluation  Strategy  to  permit  accurate  development  of 
corrective  action. 

2.  A  comprehensive  CTE  program  is  necessary  to  meet  all*  DES 
classroom  evaluation  responsibilities.  Use  of  a  single  type  of 
evaluation  instrument  for  all  evaluation  requirements  is  too  in¬ 
efficient.  It  would  constrain  DES's  evaluation  program  because  of 
limited  personnel  resources  and  also  provide  inadequate  information 
for  decision  making  on  curriculum  issues.  The  adoption  of  two 
types  of  classroom  evaluations  will  make  better  use  of  available 
resources  and  meet  most  of  DES’s  classroom  evaluation  responsibili¬ 
ties. 


3.  The  program  proposed  in  Chapter  4  will  meet  DES  require¬ 
ments  for  classroom  evaluations  with  the  resources  currently 
avai 1  able. 

4.  The  proposed  CTE  program  has  not  been  validated;  there¬ 
fore,  a  validation  phase  should  be  incorporated  in  its  implementa¬ 
tion  schedule  to  allow  "fine  tuning"  of  the  program  procedures  and 
checklist. 


.  - •, 


RECOMMENDATQN 


It  is  recommended  that  the  Directorate  of  Evaluation  and  Stand¬ 
ardization,  USAAVNCs 

1.  Implement  the  proposed  Classroom  Training  Evaluation  Pro¬ 
gram  outlined  in  Chapter  4. 

2.  Validate  the  program,  during  implementation,  to  evaluate 
the  effectiveness  of  the  program  in  meeting  DES’s  classroom 
evaluation  needs.  If  successful,  the  program  can  be  adjusted  to 
correct  inefficiencies  prior  to  final  adoption  by  the  directorate. 
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